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I.  INTRODUCTION 


This  report  is  a  summary  of  the  major  research  results  obtained  under 
Contract  N00014-78-C-0443.  Work  under  this  contract  was  direced  at  a 
general  class  of  Linear-Quadratic-Gaussian  control  problems  herein  called 
"cumulant  control"  problems. 

Over  the  past  few  years,  Liberty  and  colleagues  [1],(2]  and  [3]  have 
investigated  probabilistic  questions  surrounding  the  random  performance 
measure  normally  associated  with  linear  stochastic  control  systems.  Early 
work  was  aimed  at  finding  an  analytical  means  of  evaluating  the  performance 
of  such  systems  [3],  Results  of  this  effort  consisted  of  a  solution  to 
the  "performance  analysis"  problem.  These  results  allow  the  determination 
of  complete  statistical  descriptions  of  system  performance  without  requiring 
stochastic  simulation. 

Work,  immediately  preceeding  that  reported  here,  was  directed  at 
finding  statistical  descriptions  of  performance  for  design  purposes  [4], 

This  development  was  a  natural  analytical  extension  of  the  performance 
analysis  results.  It  led  to  a  promising  analytical  structure  for  design 
as  well  as  a  demonstration  of  the  existence  of  control  laws  that  achieve 
less  variance  of  performance  than  the  classical  Linear-Quadratic-Gaussian 
controller. 

The  original  statement  of  a  "cumulant  control  problem"  was  contained 
in  [4]  but  not  actually  solved.  Despite  the  promise  of  the  new  formulation, 
it  was  not  clear  how  to  actually  solve  a  cumulant  control  problem  involving 
statistical  parameters  otheT  than  the  mean.  The  primary  objective  of  the 
research  summarized  here  was  to  resolve  the  principal  difficulty  blocking 
solution  of  problems  in  the  cumulant  class. 

Loosely  speaking,  the  difficulty  involves  enforcement  of  an  "admissi¬ 
bility"  constraint  on  the  set  of  control  actions  over  which  optimization 
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of  cumulant  indices  of  system  performance  is  carried  out.  In  this  class 
of  stochastic  optimal  control  problems,  performance  indices  consist  of 
linear  combinations  of  mean-conditional-cumulants  of  a  quadratic  per¬ 
formance  measure  attached  to  a  linear  stochastic  system.  The  investiga¬ 
tion  was  carried  out  for  discrete  time  systems  over  a  finite  time  inter¬ 
val.  The  resulting  finite  dimensional  setting  removed  difficulty  en¬ 
countered  in  the  continuous  time  case  in  seeing  the  essence  of  the  ad¬ 
missibility  enforcement  question. 

Mean-conditional-cumulants  appear  in  this  work  as  Et*^)  where  tca|N 
is  the  a-order  annul ant  of  performance,  conditioned  on  system  output 
observations  from  time  0  to  time  N.  Linear  combinations  of  these  objects 
are  to  be  minimized  over  all  control  actions  in  an  "admissible"  class. 
Essentially,  to  be  admissible,  a  control  action  must  be  determined  by  a 

physically  realizable  operation  on  system  output  observations.  The  key 

* 

result  reported  here  is  the  discovery  of  a  set  of  random  variables  {<a} 
with  the  property  that 


E(ica>  '  ^dN1' 


(1) 


It  can  be  shown  that  constrained  (with  respect  to  admissibility)  optimiza 
tion  over  linear  combinations  of  the  latter  objects  is  equivalent  to  con¬ 
strained  optimization  over  mean-conditional-cumulants. 


I  • 
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II.  SYSTEM  DESCRIPTION 


Let  P  denote  the  set  of  integers  {0,1,..., N}  and  let  Rq  denote  the  q- 
fold  Cartesian  product  of  the  real  line.  In  this  work,  several  vector¬ 
valued  random  sequences  will  be  defined  on  the  discrete  time  interval  P. 

It  will  be  understood  that  all  expressions  containing  these  random  sequences 
hold  on  the  interval  P  unless  otherwise  specified. 

Consider  the  pro blew  of  controlling  the  noisy  system  described  on  P 
by 

x(k*l)  -  A(k)x(k)  ♦  B(k)u(k)  ♦  £(k)  (2) 

and 

z(k)  -  C(k)x(k)  ♦  9(k)  (3) 

where  the  state  x(k)eRn,  the  control  action  u(k)eRm,  and  the  observation 
z(k)eRr.  The  initial  condition  for  (2),  x(0),  is  assumed  Gaussian  with 

mean 

x0  -  E(x(0)}  (4) 

and  covariance 

r0  -  E{[X(0)  -  xo][x(0)  -  x0]T},  (5) 

T 

where  E{*}  symbolizes  the  expectation  operation  and  (  )  denotes  matrix 
transposition. 

Let  the  observation  noise,  6,  and  the  state  process  noise,  £,  be 
zero-mean  Gaussian  white  processes  with 


E{£(k)9T(*)}  -  0, 

(6) 

E{{x(0)  -  x0]£T<*)}  -  0, 

(7) 

E{[x(0)  -  x0]9T(f)}  -  0, 

(8) 

E{£(k)£T(f)}  -  H(k)6u, 

(9) 

E{0(k)9T(£) }  -  0(k)«w, 

(10) 
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where  0(k)  and  S(k)  are  positive  definite  and  positive  semi-definite  on 
P.  Here,  <5^  is  the  Kroneker  delta. 

Attach  to  (2)  a  performance  measure ,  J,  defined  by 

I  [xT(k)QCk)xCk)  ♦  uT(k-l)R(k-l)u(k-l)],  (11) 

k«l 

where  the  weighting  matrix,  Q(k),  is  positive  semi-definite  on  P  and  the 
weighting  matrix,  R(k),  is  positive  definite  on  P.  Both  Q(k)  and  R(k)  are 
symmetric  matrices. 

The  control  objective  is  to  find  a  control  sequence,  u(*)»  in  a  par¬ 
ticular  class  (called  admissible)  such  that  a  certain  statistical  per¬ 
formance  index  (formed  from  mean-conditional -cumulants  of  J)  is  minimized. 
Mean-conditional -cumulants  are  introduced  in  Section  III.  The  admissible 
class  of  control  actions  is  described  as  follows. 

Let 

u(k)  «  f (k;z)  (12) 

where 

f  :PxZ[P]-*Rm 

is  an  operator  mapping  time  k  and  an  output  sequence  z(*)  to  a  control 
sequence  value  u(k).  The  set  of  output  sequences  on  P  is  denoted  by  Z(P]. 

The  admissible  class  of  feedback  control  laws  is  determined  by  f  opera¬ 
tors  that  are 

1 .  causal 
and 

2.  satisfy  the  growth  condition 

||f(k;g)  -  f(k;h)||<E||g-h||  (13) 

for  all  keP,  for  all  g,heZ[P],  and  for  some  e>o.  An  operator,  f,  is 
causal  if  whenever  g(o)  ■  h(a)  for  o<k cP,  then  f(k;g)  »  f(k;h). 
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III.  CONDITIONAL  CUMULANTS  OF  J 


Since  feedback  control  laws  are  being  sought,  it  is  desirable  to 
formulate  statistical  indices  of  J  such  that  optimization  over  these 
indices  will  directly  couple  control  action  to  past  observations.  The 
secret  to  such  a  formulation  lies  in  the  use  of  the  conditional  expecta¬ 
tion  operator,  EC'jF^}  where  F^  is  the  minimal  o-algebra  induced  by  the 
set  of  observations  {z(t) :T£[0,k] }.  By  definition,  there  are  no  observa¬ 
tions  when  k  *  -1.  When  k  ■  N,  F will  be  written  without  a  subscript.  An 
interpretation  of  F^  for  k>0  is  that  it  represents  the  minimal  information 
necessary  to  describe  uncertainty  about  x(0),  {£(t) :Te[0,k-l] },  and 
(8(t) :te[0,k]}  after  (z(r) :Tg[0,k] }  is  known. 

The  particular  statistical  indices  presented  in  this  section  are  con¬ 
ditional  cumulants.  These  objects  are  selected  bacause  of  their  inherent 
quadratic  structure.  This  property  of  cumulants  of  quadratic  performance 
measures  was  first  observed  by  Liberty  in  [1]  and  exploited  in  [2],  [3] 
and  [4].  Optimization  over  quadratic  performance  indices  will  lead  to 
linear  control  structures. 

It  is  necessary  at  this  point  to  introduce  some  notation.  It  is  well 
known  that  E{x(k)|F}  is  the  smoothed  least  squares  estimate  of  x(k)  which 
will  be  denoted  by 

£(k  |N)  ^  E{x(k) j  F) .  (14) 

Let  the  smoothed  estimate  error  covariance  kernel,  rs(k,£)  be  defined 
as 

r#(M)  £  E{[x(k)  -  S(k|N)][x(*)  -  x(£(N]T| f  }, 

•  E{i(k|N)XT(*|N)|F},  (15) 

where 

x(kjN)  ft  x(k)  -  *(k|N),  (16) 
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is  the  error  in  the  smoothed  estimate  of  the  state  vector.  Then,  if  the 
linear  Gaussian  assumptions  of  equations  (2)  through  (10)  are  satisfied, 
and  if  the  control  action  u(«)  is  admissible. 


E{rs(k ,1)}  «  rs(k,i). 

That  is,  rg(k,£)  is  not  random. 

Next,  define  the  iterated  kernel,  rg(k,£),  as 
N 

r“(M)  $  l  r  (k,cr)Q(0)ra"1(cr,£),  a>l, 
a=l  s  5 


(17) 


where 


Also  define 


r*(k ,t)  b  rs(k,£). 


(18) 


(19) 


H  i  (k  I N)  *  l 
a  l  lml  s 


-  ^“'1(k,k)Q(k)x(k!N),  a>l. 


(20) 


It  can  be  shown  that  the  conditional  cumulants  of  J  can  be  expressed  as 

N 

<1  -  l  TR[T  fk,k)Q(k)]  ♦ 

1|N  k»l  s 

N  W 

*  l  [XT(k|N)Q(k)X(k |N)  ♦  uT(k-l)R(k-l)u(k-l)] 
k«l 


and 


,  N 

<a|N  »  (a-l)!2a‘l  l  TRtI^(k,k)Q(k)] 


♦  a.'2a  l  xT(k|N)Q(k)n  ,(k|N),  a>l,  (22) 

k-1  a"1 

where  TR[*]  denotes  the  matrix  trace  operator.  Equations  (21)  and  (22) 
are  single  summation  expressions  of  the  conditional  cumulants. in  terras  pf. 
dynamical  (but  physically  unrealizable)  variables  x(*iN)  and  na  ^’(N). 
Note  that  both  (21)  and  (22)  contain  uncontrollable  terms. 
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IV.  THE  CUMULANT  CONTROL  PROBLEM  CLASS 

The  conditional  cumulant  expressions  given  by  (21)  and  (22)  can  be 
placed  under  expectation  to  yield  interesting  statistical  parameters  of  J. 
They  are  interesting  because  they  have  a  quadratic  structure  in  the  dy¬ 
namical  variables  x(*|N)  and  na(*[N),  and  because  these  variables  can  be 
expressed  explicitly  in  terms  of  the  observation  sequence  z(*)-  The 
precise  statistical  interpretation  of  mean-conditional-cumulants  is  given 
in  [4]. 

It  is  now  possible  to  define  a  class  of  optimal  control  problems, 
called  "cumulant  control"  problems,  in  terms  of  mean-conditional-cumulants. 

Let  A  denote  the  class  of  admissible  control  actions 
for  the  system  described  by  (2) -(10)  with  performance  measure 
J  as  given  in  (11).  Let  K  i^,  a>l,  be  conditional  cumulants 
of  J,  and  let  yfl,  be  non-negative  scalar  constants  for  a>2. 

Then,  for  each  set  of  K  scalar  constants  a  cumulant  control 
problem  is  specified  by  the  statement 

Minimize  *  j2  V«|N> 

subject  to  the  dynamical  constraints  on  x(*|N)  and  na(*lN)- 
Note  that  when  y^  =  0,  2<a<K,  the  cumulant  control  problem  specified  is 
precisely  the  minimum  mean  Linear-Quadratic-Gaussian  problem  since 

E(<i|n}  *  E(J}. 

Although  the  class  of  cumulant  control  problems  is  clearly  specified,  it 
is  not  so  clear  how  one  should  carry  out  the  indicated  minimization.  The 
specific  point  of  concern  is  enforcement  of  the  admissibility  requirement. 
This  point  is  particularly  bothersome  when  one  notes  (as  at  the  end  of 
Section  III)  that  the  conditional  cumulants  have  been  explicitly  expressed 
in  terms  of  noncausal  dynamical  variables.  The  results  summarized  in  this 
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section  resolve  this  difficulty. 

If  the  conditional  cumulants  were  expressed  in  terms  of  causal  dy¬ 
namical  variables,  which  in  turn  were  driven  by  the  observation  sequence, 
then  standard  optimization  techniques,  applied  under  expectation  to  all 
control -observation  sequence  pairs,  would  couple  the  control  action  to 
these  variables  and  an  admissible,  optimal-control  action  would  be  realized. 
Clearly,  this  is  not  the  case  for  the  conditional  cumulants.  However,  it 
can  be  shown  that  portions  of  the  conditional  cumulant  formulations  are 
zero  under  expectation.  The  portions  that  remain  contain  only  causal 
variables  and  optimization  over  these  new  objects  yields  physically  realizable 
control  action  that  solves  the  original  cumulant  control  problem. 
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V.  CONDITIONAL  CUMULANTS  UNDER  EXPECTATION 

Through  Section  IV  only  one  state  estimate  has  been  encountered, 
namely,  the  so-called  "fixed  interval"  smoothed  estimate,  x(k|N).  Its 
error  covariance  kernel  has  been  given  the  symbol  rs(k,£).  In  this  sec¬ 
tion,  several  other  well-known  least-squares  state  estimates  will  arise. 
These  are:  the  "filtered  state  estimate," 

xCk|k)  *  E{x(k)  |Fk),  (23) 

the  "one-step  predicted  state  estimate," 

X(kjk-l)  =  E(x(k)  !Fk_1 } ,  (24) 

and  the  "fixed-point  smoothed  state  estimate," 

X(T  |k)  =  E{x(t)  |Fk>,  t<k.  (25) 

The  error  covariance  kernels  associated  with  (23)  and  (24)  are  useful. 
They  are  denoted  by  Tp(k,£)  and  rp(k,£),  respectively.  Defining  a  new 
matrix 

k 

Wa-i(k,a)  ^  Z  r“'1(k,^)Q(^)rp(£,a)-lr“-1(k,k)Q(k)rp(k,a),  (26) 

(20)  can  be  rewritten  as 

N 

%  ,(k|N)  =  fi  (k|k)  *  l  W  (k,o)M(o)v(o|o-l),  (27) 

a_i  Cf=k+1 

where 

k 

Vl(k|k)  =  Z  r“'1(k,^)Q(£)3c(^!k)-|r“'1(k,k)Q(k)x(k|k).  (28) 

In  addition,  the  fixed-interval  smoothed  estimate  can  be  written  as 

N 

X(£|N)  =  X(£|k)  ♦  [  r  (£,a)M(a)v(a|a-l),  (29) 

a*k+l  p 

where  v(ojo-l)  is  the  "innovations  process".  This  separation  of  the  dy¬ 
namical  variables  (present  in  the  conditional  cumulants)  into  causal  and 
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noncausal  parts  is  the  key  to  resolution  of  the  admissibility  difficulty. 
It  can  be  shown  that  the  mean-conditional >cumulants  can  be  written 


entirely  in  terms  of  causal  variables.  These  expressions  are 

N 

E{<  ,N>  *  E{  l  TR[rF(k,k)Q(k)] 

1  k*l 

N  _  T 

+  l  [x  (k|k)Q(k)x(k|k)+u  (k-l)R(k-l)u(k-l) ] } 
k*l 

*  E tie*},  (30) 

and 

.  N  k 

EOc  ,  }  -  E( (a-1) !2a"1  l  TR[{I”(k,k)+2a  J  [r.(k.T) 

a|  k=l  t*l 


-  rs(k,T)]Q(t)r“'1(T,k)-a(rF(k,k)-rs(k,k)]Q(k)r“'1(k,k)}Q(k)] 

n  T 

♦  a!2a  l  x  (k|k)Q(k)fi  ,(k|k)} 

k*l 

=  E{k*},  a>l .  (31) 

Optimization,  in  the  cumulant  control  problem,  over  these  objects 
will  couple  control  action  to  the  fixed  point  smooth  estimate  of  the  state. 
Thus,  causal  feedback  control  can  be  realized. 
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VI.  CONCLUSION 


The  results  summarized  in  this  report  open  the  door  to  several 
avenues  of  investigation  and  should  enhance  the  applicability  of  linear 
stochastic  control  theory.  The  details  of  this  work  are  currently  being 
prepared  for  journal  publication.  In  addition  to  the  results  presented 
here,  an  analogous  investigation  has  been  conducted  for  continuous  time 
systems  and  minor  investigations  of  the  cumulant  structure  as  applied  to 
estimation  problems  have  also  been  carried  out.  The  latter  studies  have 
led  to  interesting  observations,  but  inconclusive  results.  More  research 
is  required  here.  It  would  be  desirable  to  relax  the  Gaussian  assumption 
in  this  class  of  problems,  but  to  date  no  mechanism  for  doing  so  has  been 
found  for  cumulants  of  order  two  or  more. 

Dynamical  feedback  control  laws  for  certain  cumulant  control  problems 
have  been  derived  in  recursive  form  and  will  be  published  along  with  the 
details  of  the  cumulant  class  derivation.  Properties  of  cumulant  controllers 
are  totally  unknown.  Several  subopt imal  structures  are  appealing  and  should 
be  investigated.  Some  of  these  lead  to  interesting  classes  of  Riccati-type 
equations  about  which  little  is  known  [5] . 
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